-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make recipes consistent w.r. libintl, iconv, gettext #35450
Make recipes consistent w.r. libintl, iconv, gettext #35450
Conversation
Hi @greenc-FNAL! I noticed that the following package(s) don't yet have maintainers:
Are you interested in adopting any of these package(s)? If so, simply add the following to the package class: maintainers = ["greenc-FNAL"] If not, could you contact the developers of this package and see if they are interested? You can quickly see who has worked on a package with $ spack blame acl Thank you for your help! Please don't add maintainers without their consent. You don't have to be a Spack expert or package developer in order to be a "maintainer," it just gives us a list of users willing to review PRs or debug issues relating to this package. A package can have multiple maintainers; just add a list of GitHub handles of anyone who wants to volunteer. |
@grospelliergilles can you review this PR? This PR modifies the following package(s), for which you are listed as a maintainer:
|
Closes #34541 |
@scheibelp @mwkrentel As promised. |
if "intl" in self.spec["gettext"].libs.names: | ||
env.append_flags("LDFLAGS", "-lintl") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not use:
if "intl" in self.spec["gettext"].libs.names: | |
env.append_flags("LDFLAGS", "-lintl") | |
env.append_flags("LDFLAGS", self.spec["gettext"].libs.link_flags) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If libs.link_flags
gets filled in properly for gettext
as an external, that would be a reasonable thing to do.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will add links for:
["libasprintf", "libgettextlib", "libgettextpo", "libgettextsrc", "libintl"]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears that, of packages requiring libintl
, most of them require only that and not any of the others. Indeed, putting the full list in for elfutils
actually caused an ODR violation as color_mode
is defined in libeu.a
and libgettextsrc
(IIRC). In other words, using only -lintl
when needed, rather than the full libs.link_flags
, seems generally to be the way to go.
@@ -629,7 +629,6 @@ def configure_args(self): | |||
self.with_or_without("hdf4", package="hdf"), | |||
self.with_or_without("hdf5", package="hdf5"), | |||
self.with_or_without("hdfs", package="hadoop"), | |||
self.with_or_without("libiconv-prefix", variant="iconv", package="iconv"), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the build error you see without this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If iconv
is provided by libc
and is an external, then this adds /usr
as a high-priority prefix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That sounds like it isn't a bug with libc but a bug with external packages. Shouldn't you instead check where iconv is installed and filter out system paths? spack.util.environment
has is_system_path
and filter_system_paths
functions to handle this kind of stuff.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need to double-check, but I'm moderately sure that using --libiconv-prefix
tries to add -liconv
to the library list which will fail if we're getting it from libc
. The fouling of the library search path is secondary to that.
With reference to, "Shouldn't you instead check where iconv is installed and filter out system paths?" however: if "you" is, the package maintainer, the unfortunate answer to that is, "no, because nobody else does." As a release manager who configures a certain set of externals all the time, I'm forever running into packages where some aspect of the recipe: library search paths, PATH
, or other environment settings gets tripped up by using externals. It's left to individual Spack users like me to trip over the problem, fix it in every package that has the problem and submit the PR. If there were a way to have the CI test recipe builds with dependencies configured as externals in addition to having them installed in the same Spack setup, that might help. Until then however, the burden falls on the first unfortunate soul to want to build against that dependency as an external and gets to watch everything explode.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By "you" I mean the person writing or maintaining the package.py
build recipes. From your initial reply, it sounded like instead of checking whether iconv is libc, we should instead check to see if iconv is installed in a system directory where many other things are installed.
We automatically filter system deps most of the time (anything automatically added to the compiler wrappers, for example). It's hard to automatically filter them the rest of the time though. The best we can do automatically would be to ensure that all link/include flags are sorted with system locations last. Can't remember if @scheibelp or @becker33 or @haampie ever managed to get that working, or who was working on that. I agree that it's annoying to have to do this on a package-by-package basis.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By "you" I mean the person writing or maintaining the
package.py
build recipes. From your initial reply, it sounded like instead of checking whether iconv is libc, we should instead check to see if iconv is installed in a system directory where many other things are installed.
No, sorry.
We automatically filter system deps most of the time (anything automatically added to the compiler wrappers, for example). It's hard to automatically filter them the rest of the time though. The best we can do automatically would be to ensure that all link/include flags are sorted with system locations last. Can't remember if @scheibelp or @becker33 or @haampie ever managed to get that working, or who was working on that. I agree that it's annoying to have to do this on a package-by-package basis.
We already have environment utilities provided by me to do this:
spack/lib/spack/spack/util/environment.py
Lines 76 to 80 in 9b3c4e0
def deprioritize_system_paths(paths): | |
"""Put system paths at the end of paths, otherwise preserving order.""" | |
filtered_paths = filter_system_paths(paths) | |
fp = set(filtered_paths) | |
return filtered_paths + [p for p in paths if p not in fp] |
spack/lib/spack/spack/util/environment.py
Lines 330 to 338 in 9b3c4e0
class DeprioritizeSystemPaths(NameModifier): | |
def execute(self, env): | |
tty.debug("DeprioritizeSystemPaths: {0}".format(self.name), level=3) | |
environment_value = env.get(self.name, "") | |
directories = environment_value.split(self.separator) if environment_value else [] | |
directories = deprioritize_system_paths( | |
[path_to_os_path(os.path.normpath(x)).pop() for x in directories] | |
) | |
env[self.name] = self.separator.join(directories) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's already used by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@greenc-FNAL I'll note that our compiler wrappers reorder -L
etc. so even if the configure script introduces a mixed order of system/custom paths, our wrapper should disentangle that. It would be useful to see an example failed installation and specifically the corresponding modified compiler invocations you can see in spack-cc..out
if you run the install with -d
like spack -d install ...
.
Note: changes originally made in collaboration with @marcmengel. |
@greenc-FNAL Hmm, this is not exactly the patch I had in mind. First, I was expecting the logic about Also, in the gettext patch, I'm not so sure about the separation for The main difference was RH put libc in So, the dividing line between Finally, there's a separate problem over the names of the functions as https://gitlab.spack.io/spack/spack/-/jobs/5538241 It looks like some packages want the Spack doesn't really have a good way to express this. Maybe we need I'm not sure, I haven't done a careful inventory of what other |
if name == "ldlibs": | ||
if "intl" in self.spec["gettext"].libs.names: | ||
flags.append("-lintl") | ||
elif name == "ldflags": | ||
flags.append("-pthread") | ||
return self.build_system_flags(name, flags) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e471a2c
to
b34ff9f
Compare
b34ff9f
to
bc844a2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looking good, with a couple of typos. General notes:
- In several cases this just swaps out an explicit dep on
libiconv
foriconv
virtual, which seems like a good thing to do in terms of linking consistency (although somewhat a separate concept to the core issue that this PR tackles) - I'm wondering if
libintl
should be treated likeiconv
a virtual that may be provided by libc. I think that is the case but it is not modeled that way, so the libiconv/libintl handling differs
As a side note, it would be nice to add a configure_iconv_prefix_arg
method to all implementations of the iconv
virtual, which would generally avoid repetition of the following code block:
if spec["iconv"].name == "libc":
args.append("--without-libiconv-prefix")
elif not is_system_path(spec["iconv"].prefix):
args.append("--with-libiconv-prefix={p}".format(p=spec["iconv"].prefix))
That doesn't have to be done in this PR, but I'll be considering how to refactor this.
@@ -55,7 +55,7 @@ class PyOnnxRuntime(CMakePackage, PythonExtension): | |||
# https://github.com/cms-externals/onnxruntime/compare/5bc92df...d594f80 | |||
patch("cms.patch", level=1, when="@1.7.2") | |||
# https://github.com/cms-externals/onnxruntime/compare/0d9030e...7a6355a | |||
patch("cms_1_10.patch", whe="@1.10") | |||
patch("cms_1_10.patch", when="@1.10") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It appears this is a duplicate package: ideally this could be split off into another PR (which removes one of the duplicates and updates dependencies accordingly).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looking good, with a couple of typos. General notes:
- In several cases this just swaps out an explicit dep on
libiconv
foriconv
virtual, which seems like a good thing to do in terms of linking consistency (although somewhat a separate concept to the core issue that this PR tackles)
I believe this is necessary for consistency: if libc
is selected as the required provider for iconv
, then packages in the DAG with explicit dependencies on libiconv
could cause problems.
- I'm wondering if
libintl
should be treated likeiconv
a virtual that may be provided by libc. I think that is the case but it is not modeled that way, so the libiconv/libintl handling differs
I think this was deliberately not done because gettext
may be required for non-libintl
reasons. It is perfectly reasonable to use libc
for libintl
even in the presence of gettext
in the DAG.
As a side note, it would be nice to add a
configure_iconv_prefix_arg
method to all implementations of theiconv
virtual, which would generally avoid repetition of the following code block:if spec["iconv"].name == "libc": args.append("--without-libiconv-prefix") elif not is_system_path(spec["iconv"].prefix): args.append("--with-libiconv-prefix={p}".format(p=spec["iconv"].prefix))
That doesn't have to be done in this PR, but I'll be considering how to refactor this.
This is the "modern" way for the autotools
m4
macros to handle this, but it is not the case for older autotools
versions. In other words, the implementations of configure_iconv_prefix_arg()
would have to consider autotools
versions. Even that may not be comprehensive due to package-specific implementations of the handling, but I haven't done a survey.
The last commit to libxpm packages.py breaks it on RHEL8. Still compiles on fedora. Error message is that it can't find libintl:
|
Reverting the change fixes it |
@espentrydal thanks for the info: I can investigate this later this week. I would have assumed that the before/after for that package were equivalent methods of conveying the libintl dependency so I want to try to reproduce your issue |
@espentrydal @scheibelp: I guess the initial line in env.append_flags("LDFLAGS", "-L{0} -lintl".format(self.spec["gettext"].prefix.lib)) It might also be a good idea to have a look at #29698 which might be helpful when libxpm itself is an external package from a Linux system which also does not add a libintl into the build of dependent packages. |
@espentrydal Can i ask you to put in a new issue for this, please? This wasn't caught by the CI because it seems to be very platform-specific, so please provide full details of your system and minimal reproducing command(s). |
I came here to post that same error. #36568 works for me, but this PR broke libxpm for me. |
extlib_bits.append("-lintl") | ||
env.append_flags("EXTLIBS", " ".join(extlib_bits)) | ||
if not is_system_path(spec["gettext"].prefix): | ||
env.append_flags("CFLAGS", spec["gettext"].headers.include_flags) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This drops optimization flags :(
if self.spec["iconv"].name == "libc": | ||
args.append("--without-libiconv-prefix") | ||
elif not is_system_path(self.spec["iconv"].prefix): | ||
args.append("--with-libiconv-prefix={p}".format(p=self.spec["iconv"].prefix)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have looked at the configure and mono
does not seem to use configure arguments --without-libiconv-prefix
or --with-libiconv-prefix
. When it searches for libiconv
it uses default flags. If we want to change the path for libiconv
, we have to use CPPFLAGS
or LDFLAGS
environment variables.
Following
discussion re
#34541 and friends, I noted that I had a fork with comprehensive changes
purporting to solve the whole mess with
iconv
/libiconv
/libc
,libintl
/libc
, andgettext
, and all the dependents thereof.This set of changes not only gives consistent build stack behavior with
respect to the above, but also works with externals.